1 Data

A total of 302 participants each completed all 12 estimation questions for a total of 3624 estimations.

participant_id q_id creature dataset scale response qtext true_value closest_pt_value
0f7fc4ff03ad53f80b84cc62c1644648 Q0 ewok dataset1 log2 10000 Between 30 and 40 ABY, how does the population of Ewoks change? NA NA
0f7fc4ff03ad53f80b84cc62c1644648 QE1 ewok dataset1 log2 500 What is the population of Ewoks in 10 ABY? 481.62 445.48
0f7fc4ff03ad53f80b84cc62c1644648 QE2 ewok dataset1 log2 24 In what ABY does the population of Ewoks reach 4,000? 28.45 24.00
0f7fc4ff03ad53f80b84cc62c1644648 QI2 ewok dataset1 log2 16 How many times more Ewoks are there in 40 ABY than in 20 ABY? 10.69 11.15
0f7fc4ff03ad53f80b84cc62c1644648 QI3 ewok dataset1 log2 6 How long does it take for the population of Ewoks in 10 ABY to double? 6.25 4.00
0f7fc4ff03ad53f80b84cc62c1644648 QI1 ewok dataset1 log2 14848 From 20 ABY to 40 ABY, the population increases by ____ Ewoks. 14363.34 15517.75

2 Q0: Open Ended

q_id creature qtext true_value
Q0 tribble Between stardates 4530 and 4540, how does the population of Tribbles change? NA
Q0 ewok Between 30 and 40 ABY, how does the population of Ewoks change? NA

2.1 Linear Scale

## NULL

2.2 Log Scale

## NULL

3 QE1: Estimate a population given a year

3.1 Question

q_id creature qtext true_value
QE1 tribble What is the population of Tribbles in stardate 4510? 481.62
QE1 ewok What is the population of Ewoks in 10 ABY? 481.62

3.2 Closest Points

x y0 dataset y
3010 481.62 dataset1 445.48
3010 481.62 dataset2 466.90

3.3 Data

QE1 data set example
participant_id dataset scale response true_value deviation abs_deviation closest_pt_value closest_pt_deviation closest_pt_abs_deviation
0f7fc4ff03ad53f80b84cc62c1644648 dataset1 log2 500 481.62 18.38 18.38 445.48 54.52 54.52
0f7fc4ff03ad53f80b84cc62c1644648 dataset2 linear 500 481.62 18.38 18.38 466.90 33.10 33.10
5b16d305db2092ddae8cf1c410341f36 dataset1 log2 500 481.62 18.38 18.38 445.48 54.52 54.52
5b16d305db2092ddae8cf1c410341f36 dataset2 linear 400 481.62 -81.62 81.62 466.90 -66.90 66.90
8ab3438eb60817740bdd15143ceacd06 dataset2 log2 510 481.62 28.38 28.38 466.90 43.10 43.10
8ab3438eb60817740bdd15143ceacd06 dataset1 linear 800 481.62 318.38 318.38 445.48 354.52 354.52

3.4 Exploration

3.5 Main Take Aways

3.5.1 More varability in the estimation made on the linear scale

dataset scale variable n median iqr
dataset1 linear response 145 500 500
dataset1 log2 response 157 500 48
dataset2 linear response 157 500 400
dataset2 log2 response 145 500 12

3.5.2 Participants Anchored their Estimates

Sub idea: Participants are estimated based on the points, not based on the visually fitted trend. This is stronger in QE2.

3.6 Absolute deviation

3.6.1 True Value

scale variable n median iqr
linear abs_deviation 302 381.62 463.23
log2 abs_deviation 302 22.89 12.00

3.6.2 Closest point absolute deviation

scale variable n median iqr
linear closest_pt_abs_deviation 302 345.48 411.38
log2 closest_pt_abs_deviation 302 45.10 21.41

4 Years 10, 20, 40 (First level estimates from calculation and scratchpad)

4.1 Compare used scratchpad vs not used scratchpad accuracy for pop10_est

showed_work_cutoff count
no 152
yes 150
dataset scale showed_work_cutoff variable n median iqr
dataset1 linear no response 73 200 998.00
dataset1 linear yes response 72 500 376.25
dataset1 log2 no response 79 500 30.00
dataset1 log2 yes response 78 489 53.75
dataset2 linear no response 79 400 460.00
dataset2 linear yes response 78 500 339.75
dataset2 log2 no response 73 500 12.00
dataset2 log2 yes response 72 500 12.00

4.2 Data Setup

dataset year true_value closest_pt_value
dataset1 10 481.6152 445.4833
dataset1 20 1483.0129 1529.1847
dataset1 40 15846.3543 17046.9353
dataset2 10 481.6152 466.8964
dataset2 20 1483.0129 1288.9142
dataset2 40 15846.3543 24186.3409

4.3 Year 10

4.3.1 Data set 1

dataset scale year variable n median iqr
dataset1 linear 10 population_est 72 500 376.25
dataset1 log2 10 population_est 78 489 53.75

4.3.2 Data set 2

dataset scale year variable n median iqr
dataset2 linear 10 population_est 78 500 339.75
dataset2 log2 10 population_est 72 500 12.00

4.4 Year 20

4.4.1 Dataset 1

dataset scale year variable n median iqr
dataset1 linear 20 population_est 113 2000 500.0
dataset1 log2 20 population_est 125 1536 138.4

4.4.2 Dataset 2

dataset scale year variable n median iqr
dataset2 linear 20 population_est 89 1500 1000
dataset2 log2 20 population_est 111 1300 300

4.5 Year 40

4.5.1 Dataset 1

dataset scale year variable n median iqr
dataset1 linear 40 population_est 113 17000 1500
dataset1 log2 40 population_est 124 16384 16

4.5.2 Dataset 2

dataset scale year variable n median iqr
dataset2 linear 40 population_est 88 24000 700
dataset2 log2 40 population_est 110 24000 9612

5 QE2: Estimate a year given a population

5.1 Question

q_id creature qtext true_value
QE2 tribble In what stardate does the population of Tribbles reach 4,000? 4528.45
QE2 ewok In what ABY does the population of Ewoks reach 4,000? 28.45

5.2 Closest Points

x y0 dataset y
27 3369.38 dataset1 3010.82
25 2661.12 dataset1 3064.85
26 2994.03 dataset1 3169.40
28 3792.59 dataset1 3228.14
24 2365.86 dataset1 3774.90
30 4807.77 dataset1 5174.12
25 2661.12 dataset2 2746.12
28 3792.59 dataset2 3859.22
27 3369.38 dataset2 4099.69
29 4269.76 dataset2 4423.01

5.3 Data

QE2 data set example
participant_id dataset scale response true_value deviation abs_deviation closest_pt_value closest_pt_deviation closest_pt_abs_deviation
0f7fc4ff03ad53f80b84cc62c1644648 dataset1 log2 24 28.45 -4.45 4.45 24 0 0
0f7fc4ff03ad53f80b84cc62c1644648 dataset2 linear 27 28.45 -1.45 1.45 27 0 0
5b16d305db2092ddae8cf1c410341f36 dataset1 log2 24 28.45 -4.45 4.45 24 0 0
5b16d305db2092ddae8cf1c410341f36 dataset2 linear 25 28.45 -3.45 3.45 27 -2 2
8ab3438eb60817740bdd15143ceacd06 dataset2 log2 28 28.45 -0.45 0.45 27 1 1
8ab3438eb60817740bdd15143ceacd06 dataset1 linear 28 28.45 -0.45 0.45 24 4 4

5.4 Exploration

5.5 Main Take Aways

5.5.1 Accuracy

  • About the same on both scales, maybe a slight tendency to overestimate (maybe due to estimating the “visual trend” estimate on the log scale as indicated by data set 1?).
  • Not sure there is enough evidence to claim participants were more accurate on the linear scale, but would like to do more studies focusing on participants ability to read between the y-axis tick marks on the logarithmic scale. They could also be anchoring to the tick mark of 4096.
dataset scale variable n median iqr
dataset1 linear response 141 24 1.0
dataset1 log2 response 154 24 3.0
dataset2 linear response 155 27 1.5
dataset2 log2 response 139 27 1.0

5.5.2 Participants are basing estimates on the simulated data points and not on a visual trendline

  • but… based on the density plots above, on the log scale, participants might have a tendency to be fitting a visual trend first and then estimating as indicated by the overestimation on the log scale for data set 1.

6 QI1: Estimate an additive increase in population between two years

q_id creature qtext true_value
QI1 tribble From 4520 to 4540, the population increases by ____ Tribbles. 14363.34
QI1 ewok From 20 ABY to 40 ABY, the population increases by ____ Ewoks. 14363.34

6.1 Closest Points

dataset 3020 3040 increase
dataset1 1529.18 17046.94 15517.75
dataset2 1288.91 24186.34 22897.43

6.2 Data

  • Truncate data at 30000.
QI1 data set example
participant_id dataset scale response true_value deviation abs_deviation closest_pt_value closest_pt_deviation closest_pt_abs_deviation
0f7fc4ff03ad53f80b84cc62c1644648 dataset1 log2 14848 14363.34 484.66 484.66 15517.75 -669.75 669.75
0f7fc4ff03ad53f80b84cc62c1644648 dataset2 linear 23000 14363.34 8636.66 8636.66 22897.43 102.57 102.57
5b16d305db2092ddae8cf1c410341f36 dataset1 log2 13000 14363.34 -1363.34 1363.34 15517.75 -2517.75 2517.75
5b16d305db2092ddae8cf1c410341f36 dataset2 linear 2400 14363.34 -11963.34 11963.34 22897.43 -20497.43 20497.43
8ab3438eb60817740bdd15143ceacd06 dataset2 log2 14800 14363.34 436.66 436.66 22897.43 -8097.43 8097.43
8ab3438eb60817740bdd15143ceacd06 dataset1 linear 13500 14363.34 -863.34 863.34 15517.75 -2017.75 2017.75

6.3 Exploration

  • Quite a few responses in the 10 - 30 range??

6.4 Main Take Aways

6.4.1 Accuracy

  • Tendency to underestimate the difference (for closest point) and tendency to overestimate the difference (for true value).
dataset scale variable n median iqr
dataset1 linear response 142 15000.0 2975
dataset1 log2 response 156 14784.0 2000
dataset2 linear response 153 17000.0 11000
dataset2 log2 response 134 15445.5 8240

dataset scale variable n median iqr
dataset1 linear closest_pt_deviation 142 -517.75 2975
dataset1 log2 closest_pt_deviation 156 -733.75 2000
dataset2 linear closest_pt_deviation 153 -5897.43 11000
dataset2 log2 closest_pt_deviation 134 -7451.93 8240

6.4.2 Anchoring & Reading Points

  • For example, 16384 - 1024 = 15360, but that 15000 is still strong.
  • Data set 2 has higher estimates (aka reading the points).

7 QI2: Estimate a multiplicative change in population between two years (i.e. how many times larger)

q_id creature qtext true_value
QI2 tribble How many times more Tribbles are there in 4540 than in 4520? 10.68524
QI2 ewok How many times more Ewoks are there in 40 ABY than in 20 ABY? 10.68524

7.1 Closest Points

dataset 3020 3040 increase
dataset1 1529.18 17046.94 11.15
dataset2 1288.91 24186.34 18.76

7.2 Data

  • Truncated at 30000, due to misunderstanding.
QI2 data set example
participant_id dataset scale response true_value deviation abs_deviation closest_pt_value closest_pt_deviation closest_pt_abs_deviation
0f7fc4ff03ad53f80b84cc62c1644648 dataset1 log2 16.0 10.69 5.31 5.31 11.15 4.85 4.85
0f7fc4ff03ad53f80b84cc62c1644648 dataset2 linear 17.0 10.69 6.31 6.31 18.76 -1.76 1.76
5b16d305db2092ddae8cf1c410341f36 dataset1 log2 32.0 10.69 21.31 21.31 11.15 20.85 20.85
5b16d305db2092ddae8cf1c410341f36 dataset2 linear 500.0 10.69 489.31 489.31 18.76 481.24 481.24
8ab3438eb60817740bdd15143ceacd06 dataset2 log2 11.6 10.69 0.91 0.91 18.76 -7.16 7.16
8ab3438eb60817740bdd15143ceacd06 dataset1 linear 12.5 10.69 1.81 1.81 11.15 1.35 1.35

7.3 Exploration

7.4 Main Take Aways

7.4.1 Basic lack of understanding

  • Describe logic from calculation and scratchpad.
  • 15000 was still a common response.

7.4.2 Accuracy

  • Larger variability on linear scale.
dataset scale variable n median iqr
dataset1 linear response 145 11.70 8.5
dataset1 log2 response 156 10.69 6.0
dataset2 linear response 156 15.15 14.0
dataset2 log2 response 145 16.00 8.5

7.4.3 Common points

  • Not as strong of an anchoring sense here.
  • Appears to be reading the points.

8 QI3: Estimate the number of years necessary to double the population

q_id creature qtext true_value
QI3 tribble How long does it take for the population of Tribbles in stardate 4510 to double? 6.245448
QI3 ewok How long does it take for the population of Ewoks in 10 ABY to double? 6.245448

8.1 Closest Points

From QE1, the estimated population at 10 is:

x y0 dataset y doubled
3010 481.62 dataset1 445.48 890.97
3010 481.62 dataset2 466.90 933.79
x y0 dataset y doubled pop_diff year_diff
3014 747.52 dataset1 786.81 890.97 104.15 4
3013 668.65 dataset1 723.92 890.97 167.04 3
3016 936.72 dataset1 1073.52 890.97 182.55 6
3019 1320.97 dataset1 1074.29 890.97 183.32 9
3016 936.72 dataset2 1028.59 933.79 94.80 6
3013 668.65 dataset2 819.66 933.79 114.13 3
3017 1049.78 dataset2 809.55 933.79 124.25 7
3015 836.45 dataset2 750.03 933.79 183.76 5
3014 747.52 dataset2 735.97 933.79 197.82 4

8.2 Data

QI3 data set example
participant_id dataset scale response true_value deviation abs_deviation closest_pt_value closest_pt_deviation closest_pt_abs_deviation
0f7fc4ff03ad53f80b84cc62c1644648 dataset1 log2 6 6.25 -0.25 0.25 4 2 2
0f7fc4ff03ad53f80b84cc62c1644648 dataset2 linear 10 6.25 3.75 3.75 6 4 4
5b16d305db2092ddae8cf1c410341f36 dataset1 log2 8 6.25 1.75 1.75 4 4 4
5b16d305db2092ddae8cf1c410341f36 dataset2 linear 12 6.25 5.75 5.75 6 6 6
8ab3438eb60817740bdd15143ceacd06 dataset2 log2 7 6.25 0.75 0.75 6 1 1
8ab3438eb60817740bdd15143ceacd06 dataset1 linear 5 6.25 -1.25 1.25 4 1 1

8.3 Exploration

8.4 Main Take Aways

8.4.1 Accuracy

  • Larger variability on linear scale.
  • More accurate on log scale??
dataset scale variable n median iqr
dataset1 linear response 136 5 5
dataset1 log2 response 144 5 2
dataset2 linear response 147 8 6
dataset2 log2 response 136 6 2

8.4.2 Common points

  • Strong anchoring at 5 and 10.